[Model] Fix hunyuan-vl shape mismatch#31403
Conversation
Signed-off-by: wangli <wangli858794774@gmail.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request addresses a shape mismatch in HunYuanVisionAttention by correctly reshaping the attention output before the final projection. The fix appears to be correct given the context of a 4D tensor output from the attention layer. I have one suggestion to make the implementation more maintainable and robust.
| out = out.view( | ||
| x.size(0), | ||
| -1, | ||
| self.num_attention_heads_per_partition | ||
| * self.hidden_size_per_attention_head, | ||
| ) |
There was a problem hiding this comment.
For better maintainability and readability, you can simplify the calculation of the last dimension in the view operation. Instead of re-calculating the partitioned hidden size from the number of heads and head size, you can directly use the shape of the weight from the subsequent projection layer o_proj. This makes the code more robust to future changes as it directly references the expected input dimension of the next layer.
out = out.view(
x.size(0),
-1,
self.o_proj.weight.shape[1],
)
| out = out.view( | ||
| x.size(0), | ||
| -1, | ||
| self.num_attention_heads_per_partition | ||
| * self.hidden_size_per_attention_head, | ||
| ) |
There was a problem hiding this comment.
After mmencoderattn, we got the out shaple like :[B, S, N, D], We should merge the multi-head before entering the O matrix.
Not really, we will automatically reshape output to make sure it align with q, k, v input:
vllm/vllm/attention/layers/mm_encoder_attention.py
Lines 160 to 177 in 8711b21
So I think we won't hit the shape mismatch issue here. 🤔
There was a problem hiding this comment.
So, it seems the issue for ascend, I'll have a further check
There was a problem hiding this comment.
Thx, Confirmed it is the issue for forward_oot vllm-project/vllm-ascend#5443
2 participants
Purpose
After
mmencoderattn, we got the out shaple like :[B, S, N, D], We should merge the multi-head before entering the O matrix.Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.